Fix WIRE-385 stale outbound operator rosters - #603

Review — WNS-34 / WIRE-385

Verdict: the mechanism is correct and the change is close to minimal. One gap, on the Solana side, not in this PR.

Mechanism verified

apply_context::exec() runs for (ordinal : _inline_actions) trx_context.execute_action(ordinal, recurse_depth + 1) — each sibling's entire subtree completes before the next starts, so slashop/termcheck have mutated sysio.opreg before finishadv reads it.

Depth is fine: chkcons(0) → advance(1) → finishadv(2) → queueout/buildenv/accrueepoch/rcrdbatch(3), and all four send zero inline actions, so 3 is terminal against the limit of 4. Keeping flushwtdw/drainfwq/payepoch as siblings at depth 2 is the right call — theirs are the deep subtrees. chkcons is never sent inline.

On minimality

Nothing here looks cuttable:

• The continuation is unavoidable. CertiK's cheaper suggestion (exclude the local to_slash list) covers slashes only — terminations are decided inside opreg::termcheck, and flushwtdw can drop collateral below minimum; neither is predictable from advance. And the slide both reads opreg and writes epoch state, so it can't move into msgch::queueout instead.
• The withhold tightening isn't scope creep. Pruning newly creates short groups, and a published short group lowers the outpost quorum denominator — ETH's (groupSize + 1) / 2 on a group of 2 is 1, i.e. one operator settles alone.
• Future-seat repair can't be dropped either; without it the tightened withhold wedges the window permanently on the first unrepaired hole.

Worth recording: the primary WNS-34 path never withholds at all. A non-canonical-delivery slash targets an expiring-group member, which is erased wholesale — no hole, complete window, fresh roster published. Holes only arise from out-of-band removals (chalg::slash, opreg::terminate, withdrawal-driven eligibility loss).

Gap: the Solana outpost has the defect #198 fixes on Ethereum

wire-ethereum#198 exists because "applying a newer OPERATORS status did not remove the address from cached batch-operator groups, and admission trusted that cached membership." That is true verbatim of Solana, and there is no Solana companion for WIRE-385.

• liqsol-core/src/instructions/opp/epoch_in.rs:75-81 — Phase B admits on group membership only; the file has zero occurrences of status / operators / OperatorMapping.
• liqsol-core/src/instructions/opp/inbound.rs:470 — handle_operators assigns registry.operators and never touches registry.groups.

So whenever this PR withholds an incomplete window, Solana retains its previous window and keeps admitting the removed operator's epoch_in deliveries, where ETH (post-#198) clears the seat. Narrower than the headline finding — it needs an out-of-band removal plus a pool too small to repair — but it is exactly the residual case #198 was written for, and Solana is uncovered.

The green e2e doesn't clear it: run 34154585838 left BRANCH_WIRE_SOLANA empty, so it floated to the manifest default and ran the unfixed program, and no flow exercises a removed operator still delivering.

Nits

1. sysio.epoch.cpp:832 still prints sysio.epoch::advance: from inside finishadv; the sibling withhold message at :995 was updated.
2. The comment above drainfwq (:661-670) reads as two stacked leftovers from the move, and "It needs the new epoch index" is confusing when the action takes std::make_tuple(). The point worth stating is that state_tbl.set at :652 now runs before these sends so they observe the new index.

Non-blocking: "emissions retain their … inline depth" in the description is loose — accrueepoch/rcrdbatch moved depth 2 → 3; only payepoch held depth. Safe (verified neither sends inlines), but worth correcting for anyone auditing depth headroom.

Checks run

• sysio CI 34154430730 green on the exact head c412b58e86 (gcc/asan/ubsan/asserton/ubuntu24 — covers contracts_unit_test).
• e2e 34154585838 pinned sysio c412b58e86 + ethereum 10993cad32, 14/14.
• finishadv_rejects_direct_calls covers both the auth and the get_sender() guard; confirmed get_sender() returns the creator action's receiver, so the self-inline check is genuine.

Thanks for the detailed review. I addressed the two SYSIO nits in follow-up commit 902073f:

• the short-tail log now identifies sysio.epoch::finishadv;
• the deferred-maintenance comment now explains that state_tbl.set makes the new epoch visible before drainfwq, and the PR description records the actual depth change for accrueepoch/rcrdbatch.

I also traced the Solana concern against the exact revisions. The cited membership-only behavior predates SOL-377. Commit b8fae4ee added the live ACTIVE + batch-role roster check, and the current origin/next head eaebfc6 additionally scopes the seat to active_group_index. Even the earlier e2e revision 1c3f3dd already contains b8fae4ee, so no WIRE-385 Solana source delta is needed.

The Ethereum companion is now at 17392d6. It tombstones revoked or reassigned seats without shrinking quorum, rejects duplicate signer addresses, and requires a strict majority. Since the platform is pre-launch, I removed the no-op UUPS migration/reinitializer and retained the corrected contracts for fresh deployment.

Validation on exact heads 4cf466f / 17392d6 / eaebfc6:

• all 14 local canonical flow runners and heartbeat monitors pass, including flow-batch-operator-slashing with epoch-freeze monitoring;
• Ethereum Build and Test and artifact workflows pass;
• SYSIO CI and the remote 14-flow platform run are still in progress.

Re-review is still requested from @heifner.

Re-review — WNS-34 / WIRE-385

My Solana finding was wrong — retracted. I read it off a local wire-solana checkout (af373e68) that predates b8fae4ee. origin/next (eaebfc63) contains it, and so does 1c3f3dd7, so the earlier e2e also ran the fixed program. check_epoch_in_signer (mod.rs:514) requires membership in groups[active_group_index] + a registry row + ACTIVE, and assert_signer_may_deliver → key_is_eligible requires every row for the key to be eligible. No Solana companion needed. Also refining my own claim: "the primary path never withholds" holds only while the surviving roster can still fill a full tail — your single-group test correctly asserts a withhold there.

Both nits are closed in 902073f, and CI is green on the exact head (34513732666).

One outstanding item: a withheld window permanently stalls the outposts on a multi-group schedule

The Solana program already names this failure as a rollout hazard (mod.rs:491-498): "groups[active_group_index] still holds the group that served epoch N-1, so epoch N's scheduled operator is refused NotActiveOperator and the roster that would authorise it can never land — it is inside the envelope being refused. That is a permanent stall whose only recovery is a redeploy." The withhold makes that state reachable at runtime.

1. sysio.epoch.cpp:982 — active_group_index = current_batch_op_group + 1 = 1: every envelope announces the group serving next epoch. That lookahead is the precondition SOL-378 scoped admission to.
2. :741-855 — the slide and state_tbl.set run unconditionally, before the publish gate at :991. Duty rotates whether or not the window ships.
3. outpost_opp_job.cpp:52,63,95 — delivery is gated on is_elected (front group only) and pinned to read_pending_outbound(chain, current_epoch). No catch-up.

Envelope N is delivered by G_N, and the outpost learns "admit G_N" only from envelope N-1. Withhold once at N and the outpost still admits G_N when envelope N+1 arrives from G_{N+1} — disjoint by construction → refused. next_epoch_index never advances, so a later complete window fails Phase A with NonSequentialEpoch: repairing the roster does not recover it. ETH absorbs one extra epoch (_resolveChunkPosition scans all groups), and sysio.msgch.cpp:1476 requires all outposts, so the depot freezes too.

Trigger: ACTIVE batch ops below batch_operator_minimum_active on a window of ≥2 groups. Normal removals hit an expiring-group member and leave no hole, so it needs the roster genuinely below minimum — one slash or termcheck termination does it when the roster sits exactly at minimum, which is how the harness provisions and a plausible 21-operator launch shape.

Not covered: contract tests can't observe outpost admission; flow-batch-operator-slashing runs batch_op_groups = 1 (index falls back to 0, same group every epoch); flow-batch-operator-termination provisions a 10th operator, so ACTIVE returns to 9, the tail fills and nothing is withheld.

Refusing to publish a short group is right — master's alternative shrinks the quorum denominator. But the trade is fail-open → fail-closed-and-unrecoverable, and the description ("withholding … until the roster is repaired", "Current-group vacancies require roster recovery") promises a recovery the mechanism can't deliver.

Suggested direction: don't let duty advance past the group the outposts were last told about — commit the slide only when the previous window was published. The first withheld epoch is safe (that duty was already announced); freezing from the second keeps a full-size group serving, denominator intact, channel live until the roster is repaired. Failing that, say plainly in the deployment notes that a below-minimum roster on a multi-group window is terminal for the outposts.

Otherwise clean

emission_amount types line up; the reordered flushwtdw/drainfwq only read current_epoch_index (set at :652 before the sends) and queue no attestations; require_fresh_roster asserts the property CertiK actually raised.